economic and social order can be all things to ail people. Like foreign aid or 
• welfare, educational innovation argues that if the necessary -changes are impractical 
because they would upset too many . traditions, then at least school districts can 
be provided with revenue to reassure them that they are grappling with important 
issues. 122 

We don't know how far the metaphor bf ESP as revenue sharing can or should be 
taken. Given the literature on educational innovation, we suspect is is a useful 
analogy and that an analysis of many innovations would reveal a similar pattern. 
Nor do we want to argue that a legitimate function of education may be stability 
rather than change, for that argument is beyond the scope of this report. It 
seems clear, however, that given the current relationship between grantors and 
grantees, and given the institutional parameters within which schooling operates, 
and given the institutional parameters within which schooling operates, revenue 
sharing may be all that we can expect from many educational innovations. If so, 
we should reduce the rhetoric about change and innovation and acknowledge instead 
the more obvious role of federal government attempts at funding change as simply 
another means of allowing the system to operate in variations of thfe way it has 
operated all along. By acknowledging this role we might, at least, reduce the 
tendency for the local school district to be the major focus in searches for reasc-ns 
things don't work—and that would be an accomplishment; ' /' 

While we have spoken to the "what" and the "why" of the ESP, we have yet to 'detail 
what the Jefferson case can tell us about change efforts—their conceptualization 
and implementation. In the next chapter, we cover some implications of the 
Jefferson ESP for future attempts at planned change. . >■ ■ 
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The notion of social policy as symbolically reassuring is discussed 
at length in M. Edelmann, The Symbolic Uses of! Politics , University of Illinios 
Press, Urbana, 1964. See also Martha Derthick, N<fo Towns in Town. The 
Urban Institute, Washington, 1972. ' / 
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EX. IMPLICATIONS 



, The Experimental Schools Program represented a new approach by a federal 
agency in funding programs of planned change in a local sghool district. * * 
Previously the federal governrrtent's apjproach to funding reform programs 
had been (to earmark monies for change in one or two elements withlii^he-----^ 
'larger system (e\g. , a reading program, a math curriculum, or new staffing 
arrangement). The impetus or specific change most often'emanated from the ' 
funding agency, and the contract usually extended over a short period. The 
ESP differed from previous approaches- in several respects. First, the -program 
spanned apLve year period— a sizeable slice of time in the annals of government 
funding. Second, an average of one million dollars per year for five years 
represented a substantial increase in financial assistance by a goverment 
agency for a program of educational reform. Third, systemic change was to occur; 
that is, most of the Elements within the school system were to undergo change 
simultaneously. Fourth, decisions about the specific changes that were to 
occur in the several elements were to originate with the local district. Fifth, 
the federal government, through its funding agency, was to assume a stance of 
minimal intervention in the design of the program and aiso in its implementation. 1 4 

Although change did ocpur in the Jdfferson School District during project years, 
a^-has been explicated in this repoit, it should be quite clear that no systematic 
plan emerged to carry out the grandiose program of reform as conceived by the 
writers of the proposal and approved by the funding agency. There are a myriad , 
of more on less legitimate reasons or explanations why. massive, systemic reform 
did not occur, and this chapter deals with those which appear to be most salient. 
Some factors for not adopting and/or implementing the ESP on a districtwide basis 
can be explained by a failure/of either the local district or the funding agency to 
take them into consideratiAiu Other explan^j^fall into the categories lp£ inability 
to exercis^ controFor disirolination to exercf^ ca^trol. These potential (categories 
are mentioned not because data exist that permit thein to be used as analytical 
constructs operationalized in this chapter, but instead^p elevate them to al level 
of consciousness for us and interested readers. \ ' \ 

V v 1 • 

Conceptualizing systemic, comprehensive change, communicating the sense of it 
to others, deciding when a potential client had a purchase on it sufficient to^merit 
funding, deciding that a reasonable climate for change existed at the client level, 
deciding that the capacity existed at the local level in sufficient abundance to implement 
the agreed upon plan, and monitoring the implementation of such plan are tasks of 



1ft was not different from previous approaches in the sense that the local 
plaflwas to be approved by the funding agency and that some monitoring was to N 
occur during the life of the Droeram. 
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considerable magnitude. These ta*ks and more faced the planners of Experi- 
I mental Schools at the national level. To say they were merely difficult is to 
understate the case, but to designate them as impossible k may be overstating the 
case. However, when constraints at the federal level are considered, both 
externally and self -imposed constraints, substantial success appears unlikely. 
When combined with constraints th?tt existed in the Jefferson School District, 
successful implementation of the planned project appears highly improbable. 
Conditions at the federal level that impeded reform in establishing a new approach 
to promoting local change are considered next, followed by a discussion of problems' 
and issues that militated against systemic, comprehensive change in the Jefferson 
School District. Please note that it is often the interaction of the constraints 
which prevents or hinders successful solutions. 



A. Constraints at the Federal Level 



Several constraints were operating at the federal level which militated against 
the likelihood that a comprehensive, systemic program of planned reform could - 
be mounted and sustained. Some of the factors that had long hindered local change 
also sfeemed to inhibit federal officials in working out a new plan for funding a 
change program. Federal officials needed to conceptualize what was meant by 
comprehensive, systemic change and to communicate this to potential^ clients and 
significant others. They needed to conceptualize the social conditions cohducive 
to change and to determine when these conditions existed at the local level. " Jhey 
needed to establish clear and consistent policies as to how the federal and local \ 
agencies were to relate, which also included the establishment .of mechanisms and 
procedures for resolving conflict with focal agencies. Operationally, federal 
plam^er^sWd supervisors faced such constraining factors as minimal time to plan 
and ntaiint the program, insufficient resources to build and maintain a staff capable 
of supervising a program of such magnitude as well as severely limited resources 
for staff travel to the various ESP sites, almtost constant turnover in the staff, % ^ 
and a lacfk of continuity in priorities by major policy makers. 



'Although it was the task of local educational agencies, to plan the specific changes 
they were to adopt and implement,, federal officials were charged with laying out 
thef parameters of comprehensive change, making related decisions, and allocating 
funds to the Local Education Agency (LEA) as well as monitoring local expenditures. 
There were no successful models for change (that is, cooperative ventures between 
a funding agency and a local school district) to sepve as a point of departure. Pre- 
vious ventures'in school reform did not meet with much success and the picture 
has hot changed appreciably in more recent years. The existence of a successful 
modej^wpuld projbably have precluded a need for the federal government td*under- 
wjrite a new model. Models fot comprehensive, systemic reform were nonexistent*. 
Insofar as we are aware* there had been only one previous attempt at promoting 
Amprehensive change in schools by an outside agency. The Fojd Foundation had 
iunded some comprehensive change programs in the early mid-sixties. Some 
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problems had been identified but were of mTnimal or no assistance to the planners 
of ESP since the results had not been written up and published. 

The notion of providing funds to school districts to bring about extensive reform 
as alluded to in the proposal rhetoric and then watching carefully to see what 
happens seems to be rather fundamentally at odds with what is known about how 
and why change occurs. Providing fundfc for a proposed plan of reform aoi-then 
assuming a stance of minimal intervention might very well be effective under a 
set of social conditions where the incentive for change is strong and widely shared 
by the significant actors in the system targeted for change. Even then an inclina- 
tion to change is insufficient if the capability to carry out the program of reform 
is not present. It is risky to assume that the rhetoric of change cpntained in a 
proposal is reliable evidence of things present or things future. The alternative 
to making the assumption that a climate conducive to change exists is to study 
existing social conditions carefully and systematically. This calls for resources 
in time and money that usually do not accompany external funding of planned change 
and which certainly was not the case with Experimental Schools. We strongly 
recommend that this be made an integral part of any future intervention model. 
The evaluation effort might also be enhanced considerably. 



One of the important lessons to be learned from J ESP is that adequate funds are 
insufficient to induce planned reform if other conditions are not conducive to 
c han ge. It can be said that revenue to underwrite programs of change is a necessary 
but not sufficient condition* It promotes the means by which planned change can 
take place but in no way assures that the rhetoric will become reality if other 
necessary conditions areinot present. The amount of the funds does not change that 
nor does the length of the/funding period. At the same time we do not wish to 
minimize the importance tof these aspects of the intervention model because they 
do provide significant contributions. They relieve the pressure of constantly 
being in a planning mode and tiiminish search behavior for outside funds. The 
energy consumed by these behaviors can be channeled into implementing. and 
revising plans so as to reduce negative and unanticipated consequences. We see 
these elements as highly desirable in most intervention models. 

An issue that plagued the district, as well as federal officials in Washington, 
for at least the first two years of ESP wa.s that of ambiguity about comprehensive 
change. At least some of the confusion can be traced to the vagueness of the 
cjharge from USOE and NDE. It was not something peculiar to the ESP'model; the 
problem is at least as old as planned change itself. However, in this instance the - 
vagueness of the charge was somewhat intentional because specific change in the 
school district was to emanate from. the grass roots. Nevertheless, it was the 
subject of many conversations and the issue ^yas never really resolved. Apparently 
few problems surfaced during the initial and negotiation stages that were not quickly 
resolved. The broad aimed guidelines probably created more problems during 
the implementational stage and in the discussions prior to -refunding. Since the 
program design at the local level was also not very precise, few parameters 

existed for informed decision making and conflict resolution^— Consequently,- we- 

are recommending that another important step be added. Specific strategies for 
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implementing the grand program design at the LEA level should be spelled out 
in as much detail as possible before funds are granted — but only after the 
problems to be solved ha^fc been conceptualized and ends agreed upon by both 
district and federal actoi4. It seems logical to be concerned with means (solutions) 
following the previous step but before actual implementational events begin. After 
all, most of the disagreement within the local district and between the local district 
and the funding agency occurs over means and not ends. It is the tneans by which 
new programs are operationalized that create' new relationships and change the 
roles and statuses, of individuals. The major source of resistance to change is 
the risk that implem^nters^must take in assuming new roles. This is the true 
innovation and not the new materials that are introduced. They are merely the 
tools of innovation. Some change theorists and practitioners argue that the change 
rhetoric in proposals should serve only to gain social, psychological, and financial 
support and once that is achieved, it has served its purpose. They hold that a 
great amount of "wiggle room" must remain for the potential reformers and 
implementers to work out the details. We, too, are firm believers in the notion 
that action informs thinking and therefore plans must be subject to change. At 
the same time, we are advocates of the notion that plans which deal primarily 
with ends is insufficient because, they have not grappled With those issues which are 
the main source of conflict. All too often solutions (strategies for implementation) 
are laid on implementers even without ^agreement that they remotely relate to the 
problem that is to w be solved, and without any understanding of what is implied in 
changed relationships. In essence, we are advocating steps which suggest that 
more resources in terms of time and expertise be devoted to the initial v^nd 
negotiation stages. At the same time, only those potential clients would be Involved 
in the solution steps who had communicated a sense of the problem, had agreed 
internally upon ends, had been judged by the funding agency to hive a conceptual 
plan which merits further consideration and had been judged^ on the basis of 
systematically generated information, by the funding agency to possess a favorable 
climate for change as* well as the inclination and capability to implement such a 
program. The adoption of such a model should decrease the number of projects 
which meet an early demise or which undergo massive and undesirable transforma- 
tion. Presently the only means of dealing with such developments is to withdraw 
funds, and this is hot a viable Alternative because of its political infeasibility. 
We would like to speculate further by suggesting that adding the step in the model 
that deals with gathering systematic information pn capability and inclination to 
changer, as well as the step on strategies for' implementing the program (solution 
step) would have revealed that comprehensive and systemic cliange was not possible, 
or for that matter desirable, in the Jefferson district. Elaboration of this point 
is reserved for the discussion on local. constraints. ^ 

Another issue which was never resolvedy during the' J ESP was the role of the 
federal agency. The data that have been presented in this report make it quite 
clear tjiat the local clients were unsure what was expected of them in carrying 7 
out a comprehensive, systemic mandate for change. The signals from Washington 
did appear ^o be mixed. The stance of minimaf intervention on the part of federal 
officials came through clearly on the level of general intent. Although federal 
officials did not tell the district what specific changes to make, it was made 



abundantly clear from time, to time that what the district,proposed to do could 
not be done. What outwardly appeared to be freedom to implement as district 
personnel elected was oftentimes erroneous and more than a little frustrating. 
The absence of previously agreed upon clear and precise parameters that informed 
decisions precluded both sets of actors frompursuing more productive behaviors 
and which would probably have been more palatable to them. Conflict between 
Washington and the LEA was much lesa pronounced during the last half of the 
project since monitoring was almost non-existent due to scarce Resources. 

Sufficient time to plan and mount a program of considerable magnitude at the 
national level was not available to federal officials. Consequently, they could not 
give to others what was denied to them. Traditionally* time is not ustteilly con- 
sidered as a valuable resource like money and materials, but as a given. The 
truth of the matter is that time is as important as any resource — maybe more so. 
It is not sufficient to'insure success, but the chances for success should be . 
enhanced considerably if it were treated as a finite resource. An ironic 
development in ill of this is that in one respect time was thought to be an important 
factor to consider as a part of the change model. The contract covered several 
years-*-qot one or two years as was the case traditionally. Consequently, time 
and Energy could be devoted to more productive endeavors than writing proposals 
for contract renewal or for new contracts. It wotild serve no purpose in trying to 
show culpability for failing to view time as" ^resource (or any other shortcoming 
for^that matter), but what we are saying is^that time is an important and scarce 
source aad should receive due consolation. 



Experimental Schools was a program which seemingly called for a large and 
expanding staff at the federal -level. It involved millions of dollars and eventually 
25' sites. Such was not the case; the staff w^s small even when the program 
realized its full size. There was insufficient time to monitor existing programs 
because staff time was taken up with expansion problems. A related problem 
was inadequate funds for staff travel. A small staff plus minimal funds for 
staff travel proved to be a major constraint on the capabilitj^f the federal agency 
to monitor the ESP and to provide needed assistance - to people in the field. 
This doesn't appear to be. a complicated problem, but it has plagued federally 
funded educational programs for a long time. f 

Another cons training* factor was the almost constant turnover of the central 
staff in Washington. It was certainly a problem for the staff there to deal .w 4 ith, 
but it was even more of a problem at the local level. Stable relationships with 
people in the field are a must if good rapport is tobe developed and maintained. 
It was stated earlier that the aspect of change whJBh encounters the most resistance 
is the establishment of new roles and relationships, and it is no less true at 
this level. Stability not change is in order; a certain amount of stability in human 
relationships must exist if people are to take risks, and changing certainly 
involves taking risks. Staff turnover is probably related to all of the factors 
^#are discussing here, but it id closely related to inadequate staffing and con- 
stantly changing priorities, the final factor discussed under constraints at the 
federal level. 
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. Few, if any, federal educational programs enjoy high status throughout the life 
of the program and consequently, priorities may change at any moment with a 
sudden shift in personnel. The shift usually creates a new set of priorities 
which are imposed in LEA's. Confusion, sometimes bordering on chaos, Is 
the result. It was difficult to keep abreast on what the priorities were in Experi- 
mental Schools. Changes at USOE and NIE policy levels, changes at the agency 
level and differing expectations of the numerous panels of experts that descended 
on both the local project and the external evaluation team created undue anxiety 
and interfered with continuity. Presently there is a ^nagging realization among 
this evaluation staff that tfiere may not be anyone left at NIE interested in the 
results of J ESP. Ostensibly the purpose of ESP was to test^fche viability of a 
change moclel. We are not advocating the adoption of the ESP model as applied 
in J ESP for future programs, but we have attempted to highlight its strengths 
and weaknesses. We have shown what happened in one case. There are both 
positive and negative aspects to J ESP; theoretically something can be learned 
from both. A cross -site comparison of results might very well reveal that what 
has been pictured as a negative feature of the model here was a positive feature 
in another site. For example, we suggested that under certain social conditions 
and with an addition of two steps in the initiating and negotiating stages, entirely 
different results might have emerged. It is possible, even probable, that the case 
was different in another setting.. We strongly recommend that the result? reported 
here and in other ESP sites be studied for possible insights into the development of 
new collaborative change models. We further suggest that long range planning 
might partially circumvent the demise of programs before their scheduled cul- 
mination. Some things must possess enough stability to induce continuity or else 
there is no meaningful point of departure. We believe that an organization which 
purports to promote planned change in LEA's nWst develop long range plans "which 
encourage both continuity in some organizational elements and change in others. . 
Both are possible and desirable. 

As previously indicated, factors that inhibited the likelihood of a successful 
program of comprehensive, systemic change as envisioned in the rhetoric were 
not confined to the federal level. We turn to a discission of some of the more 
salient constraints at the local level. 

«p 

B. Constraints at the Local Level 



Major factors which militated against successful planned change at the local level ^ 
included the infeasibility and undesirability of comprehensive, systemic change 
ia Jefferson, the vagueness and incompleteness of the proposal rhetoric, the lack 
of a sensed substantive problem and consequently, a bsoad impetus to change, a * 
lack of incentive to change, and severely limited time in which to involve potential 
iraplementers. 
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Comprehensive, systemic change in school districts is a debatable concept. 
Organizations seek a state of equilibrium in order to sustain themselves, indeed 
a legitimate goal, and broad, sweeping change in the system militates against 
the achievement of a steady state. Schools can tolerate only so much change and 
still attend to the business of "keeping" school, and planned change programs must 
• be sensitive to this need. Even when specific but broad changes are determined at 
the local level, there are limits to what can be changed and how quickly change 
can occur. If dissatisfaction does exist with an ongoing system it is highly unlikely 
that dissatisfaction pervades institutional level management, middle level manage- 
ment, involvement of community people . in the schools, staffing arrangements, 
power and authority arrangements, instructional procedures, curricula, the 
outcom^ of schooling, the manner in which assessment occurs and the flow of 
information to inform decisions. People are reluctant to change those things with 
which th«y are satisfied even if new roles, relationships and statuses are not 
threatening to them. Where there is little impetus to change, ^here is little 
incentive ; to change. Minimal impetus to change exists when the significant local 
actors are satisfied with what is. We believe this to be generally true and We are 
further persuaded of the truth of it in the Jefferson School District. There was 
little sense of a felt substantive problem in Jefferson. A problem of adequate 
financial resources to sustain existing educational programs was evident but not 
a problem with the various programs, including the manner in which decisions 
were made. Of course, there were a few people in Jefferspn who were not pleased 
with the, status quo but no widespread dissatisfaction — not from the central 
office staff, not from the principals, not from the teachers, not from the students 
and not from the community population. In the main, existing policies, practices 
and procedures were broadly accepted. A climate conducive to sweeping change 
was not present./ A careful and systematic study of social conditions existing 
in Jefferson prior to a final funding (a step we advocated adding to future change 
models) would have revealed that a climate which facilitates massive change did 
not exist. Much of what we have report&d about social conditions 'in Jefferson 
could have been uncovered at an early point. This statement is not intended to 
be a condemnation of either federal planners or the Jefferson district but is 
raised here to show the utility of such a step. Information resulting from such 
a survey can bS useful in determining what change is feasible and desirable to 
attempt. . The gist of the charge to district personnel was that they must invoke 
change in certain elements but could specify what that change was to be within 
those elements. We strongly endorse the idea that change should be specified 
by those doing the implementing, but further believe that the change should 
include an identification of what aspect(s) of schooling are to be targeted for 
change as well as what that specific change will be. 

The proposal language was also vague and ends oriented and consequently, served^ 
mostly a ceremoniaLf unction gaining social, psychological and financial support 
for the project. A grand strategy for implementing the program design was not 
a part of the proposal nor did it develop later. Minimal time existed to solicit 
involvement of those destined to implement the project design. Little change- 
emerged that could be labeled either comprehensive or systemic. Conditions 
at both the federal and local levels had combined to render suph a development 
unlikely. * 
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Yet change did occur In the Jefferson School District during ESP as we have *1 
reported in this ami related documents. Although ESP lias not succeeded In 
reforming Jefferson as visualized— comprehensively and systemically, it did 
change much more than It would have normally changed without ESP. Just 
because numerous groups had resisted wholesale change does not mean that 
there were not some people who wanted to try some new ways of doing things. 
ESP monies made it possible to try new materials and techniques that otherwise 
would not have been possible, Although the model for change proved only minimally 
successful in accomplishing projected change, ESP was good for Jefferson. It 
served as a lubricant fbr social and bureaucratic fortes. There is an intrinsic 
quality about attempts to change. As has been previoustysexplicated, change, 
occurred school by school and teacher by teacher. When dfcs^rict people were 
ready" for change and capable of changing, they tried new programs. There was 
no systemwide strategy designed or adopted; minimal coordination occurred between 
schools and even within schools in many instances. It might be said'that what 
occurred was analogous to shopping in a supermarket. School personnel took what 
they wanted and did what they wajated with what they took. It tended to be a night- 
mare for evalua,tors (both internal and external evaluators). It was much less 
difficult to document what occurred than it was to evaluate the impact of various 
innovations. ' It is ironic that J ESP succeeded in promoting piecemeal change — the 
very thing that previous collaborative arrangements between federal agencies and 
LEA's had accomplished and which had been cited in early ESP rhetoric as a inajor 
cause for limited impact. 

Is there an alternative to the ESP model for change? We believe we haVe explicated 
r an alternative that can serve as a starting point for a new model. ESP cross-site 
information can also be used to supplement what has been learned here as well as 
other collaborative ventures that involve federal and private funding agencies and 
local school districts. 

Several years ago Smith and Keith suggested that gradualism was a viable alterna- 
tive to grand and sweeping change and we concur with their thinking. Incremental 
change is possible in most school districts while grandiose change is not. M; 
change is successful in smaller settings, it will touch people both inside and outside 
the district. It is unlikely that sufficient impetus can be mustered to change an entire 
system — whereas a sub-setting may be changed, even several elements within that 
sub-setting, provided a felt problem exists. It is less complex to get agreement on 
the functional problem, desired ends, and the means by which a solution can be 
reached if fewer people are involved and their frames of reference are not too 
dissimilar. If the solution calls for greater interdependence, coordination comes 
easier with a few people than with a lot of people. Just because planned change 
occurs in a small setting or on a small 'scale'doesn't mean that it has to be "piece- 
meal. " We also agree with tfie designers of ESP that "piecemeal" change is not 
desirable when planned change occurs that has not been thought through and planned 
carefully and systematically. Also many changes have been judged unsuccessful 
x that.did not really occur bat were assumed to have occurred. The emphasis on " 
comprehensiveness could be changed from extensiveness to intensiveness. For 
example, comprehensive change could occur in a reading program for fir^t grade 1 
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youngsters in a single classroom it multiply aspects of the program tp teach 
reading were altered simultaneously. 



We would also emphasize the point that if meaningful and effective reform is 
to occur in promoting educational reform at tfye local level, a broader social ; 
and political context than the LEAVs and funding agencies must be considered J v 
Constraints to reform outside these two settings is not within the purview of >\ 
this report; however, to attempt the development of a new collaborative change 
model without including such variables is to invite major problems and hindrances. 

C. Summary h. 



" Numerous constraints existed at both the federal and local levels that inhibited , 
an effective Implementation of the planned prograril of reform in Jefferson. 
Still It has been suggested that J ESP has been functional for the Jefferson 
School Di^rlct. It helped the district through a severe fiaancial crunch and 
permitted Individual schools and individual teachers to try some programs which " 
were beftig tried elsewhere — as well .asu»evelop new curricula. The ESP did not 
have a deleterious effect upon student jPTCoihes in general, nor did it have an 
overall positive effect, at least as w&fiOced by both internal and external 
evaluators. The ESP did have intrinsic vaiue for Jefferson in that it served 
as a lubricant for social and, bureaucratic forces. By and large, it can be 
said that Jefferson gained from the experience. 

Did J £SP serve its stated function as expressed in the grand design? We have 
concluded that generally it did not. Broad and sweeping change did not occur as 
envisioned in^the rhetoric. We have argued" that it was neither feasible or? desirable 
in this setting. 

Was the overall ESP model for inducing planned reform successful in Jefferson? 
We submit that generally it was not successful and have tried to provide empirical 
support for our case by documenting what did and did not happen. We have also 
advanced some possible explanations as to why. It was held that the ESP model 
as conceived and articulated was not really tested in Jefferson. We have argued * 
that some aspects of the model were not congruent with the reality of change and 
suggested a gradual and incremental approach but with a different interpretation 
of comprehensiveness. Finally, We advocated that new collaborative change 
mbdels should be concerned with conditions which might be manipulated that lie 
outside the LEA's and federal funding agencies. 

Was the J ESP of any value other than to the district itself? Is' what has been 
learned from this experience of any help in developing subsequent collaborative, 
change models? Has anything new been learned about planned change? We 
would respond to those questions in the affirmative. But, of course, we may be 
somewhat biased. The readers of this document are in a position to provide 
an informed response to these and other questions. 
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Methodological Issues 



r v 
Appendix A is an extension of what was introduced as a Methodological Note in 

the introduction section. In this extension we discuss a multi-level approach 

to evaluation as envisioned by the architects of the Experimental Schools Program; 

some salient .functional Issues of immediate concern in the evaluation of J ESP4 

our response to the substantive evaluation issues confronting Level II in launching 

the evaluation effort; the 4 employment of a qualitative evaluation strategy to address* 

certain types of questions; the use of a quantitative evaluation strategy to address . * 

certain types of. questions; a combination of these strategies; and finally a fev* 

comment^ on the illuminative evaluation design. § Jr 

The Multi -level Approach to Evaluation 




The architects of ESP felt that evaluation should b^ no less, comprehensive thari ; 
the proposed program change and should serve as ah analytical counterpart to " 
that program. This view of evaluation suggested that it needed to serve a broader 
function than to simply assess program impapt on targeted clientele with the 
assumption that events were occurring as planned. Evaluation was to provide 
constant process and 'product feedback about the program that had the potential 
of contributing to informed decision pairing at the local level. An internal * 
evaluation unit was to be developed (if the capacity did not already exist) to 
provide these data to local decision makers. The unit was to be an integral 
part of the local change program and responsible to the school district. The 
role and function which the Level I evaluation unit performed in the J ESP is 
presented in Chapter VH of this' report, i- 

Evaluation was to provide information related to the problems encountered in 
implementing iA>vations. Information was to be generated on many aspects 
of the change qport, thereby yielding some insight into problems associated 
with planned bhange. Impact data on targeted clientele were to be gathered, 
analyzed and interpreted. These general chkrges were to be carried out by . 
an on-site, external evaluation team (Level II). The contract t*r the external 
evaluation of the J ESP was awarded to the Northwest Regional Educational 
Laboratory. In the summer of 1971 an evaluation team was placed in the district. 
An independent unit, the team reported directly to the funding agency through 
NW^EL — not to the district. Since the external evaluation unit was not to 
influence the project, most Level II reports have not been shared with the 
district heretofore. * 

: • , f 

♦There were, a few exceptions since it was deemed necessary to legitimize ouij 
presence and establish credibility with the local sc hool people. *The few reports 
that were shared with the district' were judged jointly with the federal client ^ 
not to be in violation of the non-interference posture of Level n. Most of them 
will now be shared since the project is ended. 

> 
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The funding agency wanted to gain an unbiased view of J ESP, hence the second 
level of evaluation. Evaluation information critical to district decision makers 
was to* be fed back through Level I while information critical to the intervening 
agency was to be gathered independently and fed directly to them. 

A third source of information for thfe intervening agency was to emanate from cross- 
site comparisons of common programs, problems and issues. These data were to 
be gathered by a Level III evaluation unit which was to be housed in the offices of 
the federal agency. This evaluation unit never materialized althougb-eev^ral 
efforts were madfe to. launch such.' It seems to us, that the ESP program could 
'have been well served witksuch a capacity. The reasons that position is held have 
already beep discussed in Chapter EX. 

Immediate. Func^^al Concerns . 

Leyel I and Level II teams were on. site in the summer of 1971, but after a year no 
agreement had been reached as to how the separate evaluation units would function 
cooperatively in a single Betting.^ A plan for Level I had been approved in the first 
flUity months district prop4|pal, but the details of how the two te%ps would gather 
an$I share common data were to be worked out jointly and approved by USOE. The 
one thing that did appear to be clear wa°s that although the two evaluation teams 
were to be independent of each other in determining the questions to be addressed, 
they were no,t t%t^yrrage the district with data demands when common data could 
*be utilized.* % • 

NJ?greement had been reached between USOE ahd Level II evaluation as to what it* 
should be about, although the team had been on site for approJc^mately a year. All 
of the blame for the absence of such a plan did not rest sqlely with Level II. The 
charge for Level n evaluatipn was vague and consequently much latitude for varying 
interpretations existed as to what was expected. A new director assiimed responsibility 
for the Level H evaluation program in July of 1972 and two new team members were 
added by the end of Octobej:. Although no approved evaluation plan existed, it was 
"obvious that student achievement and attitude data would be needed. An arrangement 
on gathering and sharing thfese data was, worked out with the Level I director. A 
previous agreement existed that data from field notes would not be shared. Their 
initial budget had been predicated on the idea that Level II would underwrite their 
share of data gathering and preliminary analysis. With the reacljing of an agreement, 
it was now feasible for Level -I to proceed with their approved plan, and Level II 
could concentrate on developing an evaluation plan. * *■ 

Wq perceived our task to be two-fold. Our first priority was to come up with a 
conceptualization of the evaluation problem that confronted us before providing a 

detailed operational .plan. Following a conceptualization of the task would be the 

d e v e lopment of an operational plan including a work statement. We also believed 
that the plan must be developmental. The two staff members who had been in the 
district v the first year of the project were quite knowledgeable about the state of • 
the ESP. Their insights, coupled with what all^of us knew about reform efforts in 
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general, led us to believe that whatever plan we developed must be subject to 
change depending on the information the team came up with on events that were 
actually occurring in the Jefferson District. We had the concurrence of our 
Project Officer, but while in the throes'fif writing a developmental plan, it was 
learned that NEE was sending a panel to Jefferson to' evaluate the evaluation. 
The saga of that experience will not be detailed here, and is mentioned only 
because it was a major functional issue with which we had to grapple* It 
presented a particularly difficult situation for us becau&e we now had eight 
different agenda with which to deal; our agenda, NIE's agenda, and the agenda 
of the six different panel members. It was impossible to determine who wanted 
us to d6 what. Anyway we survived the experience and lived to see an overall 
evaluation design eventually approved by NIE. Although evaluation project 
officers changed many times in the next three and one-half years, the approach 
to evaluation ixx the J ESP did not change radically. Only the quality of our 
studies improved with the addition of Level II staff.and the flexibility and 
excellent technical assistance offered by project staff in Washington. . 

These were certainly not all of the functional problems that were encountered 
during those first few months, but the'treatment of a few of the more salient 
ones should provide a flavor of the climate we faced during those first several 
months. The next section deals with the substance of developing an evaluation 
methodology consonant with the program we were charged with documenting and 
evaluating. 



Substantive Evaluation Issues 



Through studying the Request for Proposal (RFP) for the Level II evaluation* 
and also through conversations with the federal director of evaluation for ESP, 
it was learned tha^ there were three major considerations. Firs£, the impact 
of the ESP intervention on district clients was to be assessed. Second, Level II 
evaluation was to document what-happened as the Jefferson Schools implemented 
the project. Third, the planners of ESP wanted a broad evaluation design that 
encompassed more than existing evaluation models. It was the responsibility 
of Level' II evaluators to provide the specifics. We interpreted the general 
charge to mean that answers should be sought to two basic questions: How did 
the J ESP work? How will the J ESP work? Added to the general charge from 
Washington were two other basic questions: Why does the J ESP take certain 
forms? How dtf a government agency and a locaTeducation agency work 
together to bring about planted change? 

We did not believe that very many resources should be expended in pursuing the 
impact question without first illuminating the events that were actually occurring 
in the Jefferson District. ( This posture elevated the importance of implementation 
variables by suggesting that they should be subject to empirical investigation. 
At the same time, the importance of studying outcome variables was not minimized. 
Adding the why question had the potential of helping' to explain why planned 
change occurs or fails to occur. We also believed that the role of the inter- 
vening agency added another important dimension in understanding what happened 
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and why it happened. The genesis, contact, planning, negotiating and monitoring 
stages at all levels are integral parts* of the project and must be^ understood before 
a complete portrayal can be projected. In essenqe, what we proposed to do, and 
actually did, was to provide an indepth case study of an entire school district 
utilizing both qualitative and quantitative methods of gathering ancl analyzing data. 

Since the project was to aff^pt several elements of the school system such as 
governance, community, curriculum and instruction, staffing, training, and 
assessment, it seemed to make sense to develop specific studies that were 
focussed on these elements of the system — studies that captured both process 
_and, product. Additionally; we attempted to record, analyze and interpret 
interactions between the Jefferson LEA a^nd the intervening agency. Some forty 
studies were conducted during the fgur years this particular team was on site. 
The final report also includes some data that were not Included in individual 
interim reports. To effectively address the several qufestions, it was necessary 
to conduct both qualitative and quantitative studies. Consequently, the research 
and evaluation team was staffed with interdisciplinary scholars tvith diverse 
backgrounds and training. Specific designs were developed for each study and 
are included in each interim, report* % However, an attempt is made here to 
provide some general comments on how certain questions were addressed. 

Qualitative Strategies 

Documentation was interpreted to mean that events, interactions and activities 
should be recorded and analy^d. For the most part, the whstf happened question 
as well as the why question Was pursued qualitatively. Those data are mostly in 
the form of field notes taken while visiting classrooms, attending meetings, 
conducting interviews and looking through documents. Some of the data from 
field dotes yielded quite well to quantitative analyses as in the case of studies 
in curriculum and instruction, community and governance. It was felt that 
. outcome studies would not.be particularly meaningful unless treatments were 
identified a§d described, although it was necessary to conduct some quantitative 
studies without benefit of empirical analyses of process or implementation 
variable|p"TSluch qualitative data were gathered, analyzed, interpreted and 
reported in the governance component of tto y ^tudy, the community component i 
the curriculum and instruction component, and the assessment component. In 
some cases, these data could not be tied to quantitative outcome data and in 
other cases they could be. This does not mean, however, that impact was not 
reported in all cases because it is possible to use qualitative data to assess 
impact. In some instances these evaluators were trained educational ethnographers 
and in all cases they had jsome training in field methodology. * . x 

'As an example of what is being referred to here, a study from the curriculum and 
instructional component is cited. The team wanted to get some purchase on the 
variety and extent of use of curricular programs and ins true? tional practices. The 
purpose of the study was two-fold: (1) to get some notion^of what the' district was 
implementing and (2) to later be able to study outcome variables of students in 
relation to . treatments. Classrooms on all levels were sampled by actually sitting 
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in those classrooms and recording information on the use of curricular programs 
and materials, instructional practices, teacher and student interactions and 
activities and a host of other variables. This study also served as a valuable 
information source in later tying students to program treatments. 

\ 

Our evaluation team found, in the main, that observation was most effective 
in gathering information on what was actually occurring in the project, and it 
didn't sejem to matter whether it was curriculum, instruction, community, 
governance or evaluation. The why question was most effectively addressed 
through follow-up interviews with significant actors or through formally . 
structured interviews. The impact or how well question was most effectively 
addressed with quantitative data, but only after we were able to identify the 
event, activity or interaction that would cause or in some way be related to 
the outcome. The largest single drawback to the use of the qualitative method, 
especially observation, is that it is expen$ive*to implement. At the same time, 
it is cost effective when the richness of the acquired data is considered. 

% 

Quantitative Strategies 

Quantitative data were used primarily in six ways: (1) to assess the impact of 
the overall ESP intervention on student achievement without illuminating 
implementational variables, (2) to assess the impact of specific ESP programs 
or student achievement but illuminating implementational variables with 
qualitative.data (sometimes transformed to quantitative data), (3) to assess 
the impact of specific ESP programs on student achievement while illuminating 
implementational and other independent and control variables witb quantitative 
data, but informed by qualitative data that had been gathered earlier in the 
project, (4) to assess attitudes, opinions and sentiments of parents with students 
in Jefferson schools regarding ESP thrusts and programs, (5) to assess the 
institutional character of the Jefferson staff before and during the project 
using perception data, and (6) to trace the expenditure of ESP funds during * 
the project and non-ESP fundi before and during the project. 

For the most part, we found that quantitative data were more amenable to 
answering questions of tinpact than qualitative data; it was certainly more 
efficient. However, in considering the question of impact, it should b6 
reiterated that the results were often difficult to interpret when not associated 
with process variables on which some form of empirical analysis had been done. 
Interpretation often took the .form of speculation with the advancement of alternative 
explanations -^npt uncommon when using traditional models of evaluation. 

Combining Strategies 

The strength of our evaluation design resides in the fact that several data- 
gathering and Analysis techniques were utilized. We,did not pick up on the 
polemic between the advocates of field methods and the proponents of survey 
techniques. We made the assumption that data-gathering techniques cannot 
be divorbed from* the nature of the problem. The problem under investigation 
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must dictate the method or methods of investigation; no one best method exists 
th&t can be effectively utilized in all situations. Fieldwork contributes to survey 
analyses and vice-itfrsa. We found that a combination of these strategies 
often enhances interpretation of results. We also found that fl^p utilization of 
one method often facilitates the use of another. But just as important, we also * 
found that combining techniques is sometimes the best solution. This" was done 
in community studies, curriculum anifinstructionaj studies and student impact 
studies. The specific designs of these studies can Jb? found in jnterim repots 
mentioned in this reporttaifd listed in^the ERIC system*. 
■ + 

The Illuminative Evaluation Model 

* 9 

This was a case stucty of a single educational system, the Jefferson School District, 
utilizing both field and survey methods. Much has b^eft made of the usefulness 
of illuminating both process and product variables; oi elevating implementational 
variables and subjecting thexp to empirical analysis; of the utility in appraising 
events rather than non-events; of the richness of observation and other field- 
generated date; and of the usefulness of combining evaluation methodologies. 
To our knowlttlge, this is the first time that all of these have>been successfully 
accomplished in the same investigation. The utility of this inotael in studying 
planned innovation is readily apparent. Events cannot be assumed nor is 1 it 
sufficient to/speculate about alternative explanations of what the data might be 
suggesting if innovations are to b4 understood and decisions are to be made on 
the basis of evaluation information. It is wasteful of resources to judge reform 
programs as successes. or failures on the basis of severely limigpfl information. 
This is opt the 'first time that scholars have called for^treater illumination; 
however, it may very well represent the first attemptTBpLctually illuminate a 
program of planned reform. We^ecOgnize that this evaluation effort was replete 
with problems and that we also made mistakes which we would not make again. 
It does, however, represent a good beginning in utilizing a more inclusive evaluation 
model. 

. . \ 

One final point should be made. Most evaluation projects are not funded at a level 
approaching this one* But on the other hand, that could be least a partial 
explanation of why evaluation historically has enjoyed minimal success in helping 
us to understand and find solutions to the myriad problems associated with 
planifed change, * 
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. List of Level n Interim Reports 



C ommunlty/ C omponent 



Mulhauser/ F.i, "Charrette Report/ 1 Septerfiber 8, 1972. 

♦Eixenberger, D. F. , "Community Profile of the Jeff erjson School District," 
July 13, 1973. 

Eixenberg^, D. F. , Hawley, B. , and B; Fite, "Survey pi the Parents^|the 
Jefferson School District, " November 1973. \ W 

* Eixenberger, D. F. , Hawley, B. , and B. Fite, "Impact of the Experimental 

Schools Project on Parents of the Jefferson School District," April 1974.^ 

i 

* Eixenberger, D. F. , "Formal Community Involvement in Policy Formation at 

the Board qf Education and the Education Council Since the Development 
of the Jefferson Experimental Schools Project, " July 1974. 

* Eixenberger, D. F. and J. Crist-Whitzel, "Parents and the Schools: An ExsMiina- 

tion of Consensus, Perceptions of Quality, and Involvement," January 1976, 
■ ,.. ' . * # • 

* Eixenberger, D. F. , "Report on Formal Relationships Between School and Community 

at the Building Level, " March 1976. 

i • 

Eixenberger, D. F. , "Community Participation in the Jefferson Experimental 
Schools Project, " April 1976. 

. W 

•Governance Component 6 

Mulhauser, F. , '-Tteport to Mr. J. Schiller," May 9, 1973. 

Pierce, D. , "Jefferson Education Association-Jefferson School District Relationships, 
March 1974. 

■» 

Pierce, D. , "The Fiscal Environment of the Jefferson Experimental Schools 
Project School Finance Structure, " March 1974. 



Mulhauser, F. , (with editorial revision by D. Pierce and W. Doyle), "School- 
University Relations in the Jefferson Experimental Schools Project: A 
Two -Year Review, " working paper, March 1974. 

. .a , t 

Pierce, D. , "Project Governance: A Three-Year Review, " August 1974. 
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*Pierce, D. , "Perceptions of Institutional Character, " December 31, 1974. 

♦Pierce, D. , "Jefferson Educational Association-Jefferson School District ^ 
Relationships: Impacts of the Teachers' Association Upon Governance, " 
June 1975. 

$ 

♦Pierce, D. , "Project Governance: A Fourth Year Review," September 1975. 

♦Pierce, D. , "Project Governance: A Five-Ydlfr Review, " April 1976. 

♦Donicht, K.% "Jefferson Experimental Schools Project: A Fiscal Orientation," 
May 1976. 



Instructional Environment Component 

Everhart, R. B. , "Schooling and the Socialization of Knowledge," working paper, 
August 1973* v 0 

Everhart, R. B. , Hawley, B. , McGeever; J. M. , andR.'M. Toepper, "A Survey 
of Selected Educational Practices in Seven Schools Before their Official 
Involvement in the Jefferson Experimental Schools Project, "May 1974. 

♦Everhart, R. B. , Hawley, B. , McGeever, J. M. andR. M. Toepper, "Curriculum 
and Instruction in the Mathematics, Communications, and Social Studies 
Streams in the Jeffetson Experimental Schools Project, " September 1974. 

Toepper, R. M. , "A Study of Curricular Decision Making in the Jefferson School 
District Experimental Schools Project, " December 1975. . 

Everhart, R. B, and J, M. McGeever, "Career and Personal Development Streams 
in the Jefferson School District, " January 1976. 

McGeever, J, M. , "The Instructional Expedience of Selected Students in Traditional 
%nd Innovative Instructional Settings, " January 1976. 

Toepper, R. M. and J. M. McGeever, "The Status of Individualized Instructional 
Strategies on the Elementary Level (K-6) in the Jefferson School District 
During Its Five-Year Experimental Schools Project, " April 1976. 

t, R. B. , "The Worlds of the School: Everyday Life in a Junior High School, 
•July 1976. 



Student Impact Component 

Reynolds, L. , "Report on Student Transiency in Lincoln High.School, " 
' May 30, 1973. 
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Reynolds, L. , "Descriptive Survey of Twelfth Grader's In the Lincoln and 
Jefferson High Schools, 11 June 30, 1973. ' - 



* Reynolds, L. andT. Donicht, J'Mpnitoring Report I: Changes) in Student 

Achievement in Jefferson During the First Two Years of the Experi- 
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APPENDIX C 



OBJECTIVES OF THE FIRST 30 MONTHS OF THE JEFFERSON ESP 



The final draft of the original ESP proposal: "Experimental Schools, Jefferson: 
Providing Optimum Learning Environments for 4000 Students, " an application to 
the Department of Health, Education and Welfare for an operations grant under 
provisions of the Experimental Schools Program, planning project OEG-0-71 -1214, 
project number 18040, submitted byfthe superintendent, May 15, 1971, pp. 5, 8 
and 10. 
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RELATIONSHIP OF IDENTIFIED EDUCATIONAL NEEDS 



TO, PROPOSED PROJECT OBJECTIVES 



NEEDS 



OBJECTIVES 



ALL students need to experience 

MtCCESS .); 



Students need to be active parti- 
cipants in the learning process 



Students need to develop REAL 
RESPONSIBILITY 



1. 



2; 



3, 



1. 



2. 



Decrease by 75% the number of 
students behind grade level. 
Maintain rate of progress for students 
at grgde level. 

Decrease by 25% time required by gifted 
students to complete academic assign- 
ments. 

Offer eaph student some course in which > 
he can be successful. > < ■ 

. : m . ■* 

All students in one or more classes o$ 
less than 15 students daily. ' ' 

High priority to individualized and 
independent study programs. 



1. Vocational work experience provided 
for all students K-12. 

2. Graduation requirements include at 
least one-semester of work experience 
and certify that the student could accept 
responsibility from' sources external to 
the school. 



Students need to develop a POSITIVE 
SELF IMAGE 



1. 



2. 



Programs designed for competition 
with SELF rather than with PEERS. 
Opportunities, when needed, for one -to- 
ffee pupil-teacher groupings. 



Students need relevant curriculums 
with content built 1 around student 
interests* 



New FIFTH day Interim and summer 
month curriculums built on student 
interests. 

Technical-Interdisciplinary curriculums 
relating INTEREST to academic subjects. 



Students need to be prepared for 
the FUTURE 



1. Students will participate in experiences 
preparing them for shorter HIGH PER- 
FORMANCE work weeks and be able to 
constructively use leisure time. Learn 
relevant THINKING SKILLS. 
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NEEDS 



OBJECTIVES 



Today's students nded JDEAEJSM 



Teachers and all instructional 
personnel NEED training to work 
at a higher performance level 



School buildings NEED to be utilized 
during the entire year 



1. School allows activities WITHIN . ■ 
school rather than WITHOUT, 

2. Students will demonstrate a realistic 
approach to non-mate rialjstic values. 

1. All teachers in ungraded programs 
will be trained to work adequately 
with pupils grouped in flexible, con- 
tinuous progress curriculums. 

2. Teachers will demonstrate improved 
educator-learning relationships. 

3'. Teachers will be able to accept the 
concept of differentiated assignment 
based on variable performance of 
students. 

To increase the availability of schools 
to students tty at least 33% by keeping 
them op$n in the evenings and pro- 
gramming their use during the summer 
months. 



Schools NEED to utilize staff members 
more efficiently 



The COMMUNITY NEEDS to be 
involved in school operations 



Individualized and group oriented 
curriculum NEEDS to be efficiently 
managed 



Using a formula acceptable to the 
teachers association and parents, 
increase the adujt-pupil ratio to 1:12 
without significantly increasing the 
costs. Develop adequate job descriptions 
and conduct training for these~added 
personnel. 

Establish the CHARRET.TE as a regular 
procedure for converting community 
desires into real programs. Utilize 
and train selected members of the 
community for volunteer service in the 
schools. 

The amount of time a student spends 
in school will be determined by Ifia 
performance level. The school will 
move from an emphasis of providing 
180 days of "custody" to emphasizing 
a change in the level of "performance. " 
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NEEDS 



OBJECTIVES 



The INSTITUTION NEEDS schools which 
the patrons can afford 



A cost-effectiveness formula will be 
applied to PRESENT as well as new 
curriculums utilizing the following 
formula: 

TOTAL COST X QUALITY RATIO _ c 

INSTRUCTIONAL COST 

(Coefficient) 

Those programs which can be justified 
either frrom increased QUALITY or 
decreased instructor's cost will be 
retained. Others will be modified 
or discontinued. 



There NEEDS to be a training program 
for each of the project components so 
that successful practices can be 
exported 



Two local universities will establish 
training in thi%e diminensions : N 

1 . — Incorporate information aboutproject. 

components In their regular training 
programs. 

2. Develop training for INSTALLERS 
including workshops, visitation, and 
material development so that project 
components can be successfully 
exported. 

3. Conduct mini-courses and clinics 
on project components. 



The Project NEEDS to be accounted 
in the PPBES format 



To cost account all programs in terms 
of performance objectives: 

1. For the federal budget for the 
1971-72 school year. 

2. For the TOTAL school budget of 
the project of the project schools 
for the 1972-73 school year. 



A system NEEDS to be developed to 
determine which project components 
should be exported and which promising 
programs should be imported 



To develop a quality and efficiency import, 
export model and apply this instrument 
to all components to be imported for the 
1971-72 v school year and develop the 
criteria for all components ready to 
by exported By 1973-74. ^ 
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NEEDS , 

The project NEEDS to be able to^ . 
handle visitors to the project schools 
without these visitors becoming part 
of the educational process. 



Evaluation results NEED to be used 
to continue, discontinue or modify 
project components. 



OBJECTIVES 

A television dissemination center will 
be established at a local univ^jpsity to 
( bring live signals from eighrfofcations 
in the three proximity schools. These 
would be distributed through thdjtexi st- 
ing T. V. networks. . Visitors would 
have an orientation at ^he univereity, 
view selected classes fti action,^jand 
visit one or two sites instead of all six;^ 

After the first year, projects showing 
negative evaluation valuep will Unmod- 
ified. After the second *ar f tttef 
negative value projects will be\<JiCbn- 
tinued. • * 
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